Fix git-sync rotated credentials support in Helm chart by sidshas03 · Pull Request #63276 · apache/airflow

sidshas03 · 2026-03-10T14:50:11Z

Hi team, this PR is to handle token rotation properly when dags.gitSync.credentialsSecret is used.

Problem

Right now password/token is passed as env var (GIT_SYNC_PASSWORD / GITSYNC_PASSWORD).
When secret is rotated (for example ESO GitHub App token), running git-sync container does not pick the new value until restart.

What I changed

Added secret volume mount for git-sync credentials (git-sync-credentials) in:
- scheduler
- worker
- triggerer
- dag-processor
Switched password config to file-based envs:
- GIT_SYNC_PASSWORD_FILE=/etc/git-secret/credentials/GIT_SYNC_PASSWORD
- GITSYNC_PASSWORD_FILE=/etc/git-secret/credentials/GITSYNC_PASSWORD
Kept username keys from secret as before (GIT_SYNC_USERNAME / GITSYNC_USERNAME).
Updated chart values comments + schema description.
Added/updated helm tests for scheduler/worker/triggerer/dag-processor.

Why this fix

git-sync can re-read password from file path on sync loop, so rotated token is picked without forcing pod restart.

Testing

Ran helm unit tests for the updated git-sync paths and related suites (HELM_TEST_KUBERNETES_VERSION=1.32.8), and selected tests are passing.

Ran focused Helm unit tests for the new credential-file behavior:
- test_git_sync_scheduler.py::test_should_set_username_and_password_file_env_variables
- test_git_sync_worker.py::test_should_set_password_file_env_variables_when_credentials_secret_is_configured
- test_git_sync_triggerer.py::test_should_set_password_file_env_variables_when_credentials_secret_is_configured
- test_dag_processor.py::test_should_set_password_file_env_variables_when_credentials_secret_is_configured
  Result: 4 passed
Ran broader git-sync regression subset:
- Full files:
  - helm-tests/tests/helm_tests/other/test_git_sync_scheduler.py
  - helm-tests/tests/helm_tests/other/test_git_sync_worker.py
  - helm-tests/tests/helm_tests/other/test_git_sync_triggerer.py
- Plus dag-processor git-sync checks:
  - test_validate_if_ssh_params_are_added_with_git_ssh_key
  - test_should_set_password_file_env_variables_when_credentials_secret_is_configured
    Result: 42 passed
Sanity checks:
- jq empty chart/values.schema.json (schema JSON valid)
- Python compile check for modified tests (compileall) passed

Environment note: tests were run with HELM_TEST_KUBERNETES_VERSION=1.32.8.

electrical · 2026-03-10T17:10:31Z

@sidshas03 Airflow currently uses version 4.4.2 of gitsync, according to kubernetes/git-sync#976 the feature required is in 4.6.0?
I could be missing something :)

sidshas03 · 2026-03-10T17:35:09Z

@sidshas03 Airflow currently uses version 4.4.2 of gitsync, according to kubernetes/git-sync#976 the feature required is in 4.6.0? I could be missing something :)

Good catch, thanks for pointing this out. You’re right, in git-sync v4.4.2, --password-file exists but it is not re-read on every sync. The auto re-read for rotated credentials came in v4.6.0 via kubernetes/git-sync#976. So to make this fix effective by default, we should run git-sync >= v4.6.0.

Miretpl

Overall - good idea, but we cannot break backward compatibility for Git-Sync (we still support version 3 even). I placed some idea how to do it in the comments.

Also, Git-Sync container is used in pod-template-file too (used by KubernetesExecutor). Could you add a change there too?

Miretpl · 2026-03-10T20:23:49Z

chart/templates/scheduler/scheduler-deployment.yaml

        {{- if or .Values.dags.gitSync.sshKeySecret .Values.dags.gitSync.sshKey}}
          {{- include "git_sync_ssh_key_volume" . | indent 8 }}
        {{- end }}
+        {{- if .Values.dags.gitSync.credentialsSecret }}


Suggested change

{{- if .Values.dags.gitSync.credentialsSecret }}

{{- if and $localOrDagProcessorDisabled .Values.dags.gitSync.enabled .Values.dags.gitSync.credentialsSecret }}

following git sync container falgs for scheduler

Miretpl · 2026-03-10T20:25:05Z

chart/templates/triggerer/triggerer-deployment.yaml

        {{- if or .Values.dags.gitSync.sshKeySecret .Values.dags.gitSync.sshKey}}
          {{- include "git_sync_ssh_key_volume" . | nindent 8 }}
        {{- end }}
+        {{- if .Values.dags.gitSync.credentialsSecret }}


Suggested change

{{- if .Values.dags.gitSync.credentialsSecret }}

{{- if and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled) }}

following git sync container falgs for triggerer

Miretpl · 2026-03-10T20:26:44Z

chart/templates/workers/worker-deployment.yaml

        {{- if or .Values.dags.gitSync.sshKeySecret .Values.dags.gitSync.sshKey}}
          {{- include "git_sync_ssh_key_volume" . | indent 8 }}
        {{- end }}
+        {{- if .Values.dags.gitSync.credentialsSecret }}


Suggested change

{{- if .Values.dags.gitSync.credentialsSecret }}

{{- if and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled) }}

following git sync container falgs for workers

Miretpl · 2026-03-10T20:28:29Z

chart/templates/_helpers.yaml

    defaultMode: 288
 {{- end }}

+{{/*  Git credentials volume */}}


Suggested change

{{/* Git credentials volume */}}

{{/* Git credentials volume */}}

Miretpl · 2026-03-10T20:29:40Z

chart/templates/_helpers.yaml

-    - name: GIT_SYNC_PASSWORD
-      valueFrom:
-        secretKeyRef:
-          name: {{ .Values.dags.gitSync.credentialsSecret | quote }}
-          key: GIT_SYNC_PASSWORD
-    - name: GITSYNC_PASSWORD
-      valueFrom:
-        secretKeyRef:
-          name: {{ .Values.dags.gitSync.credentialsSecret | quote }}
-          key: GITSYNC_PASSWORD
+    - name: GIT_SYNC_PASSWORD_FILE
+      value: "/etc/git-secret/credentials/GIT_SYNC_PASSWORD"
+    - name: GITSYNC_PASSWORD_FILE
+      value: "/etc/git-secret/credentials/GITSYNC_PASSWORD"


This change breaks backward-compatibility. You could prevent that by making if condition on these envs where, by default, old behaviour will be in place.

Miretpl · 2026-03-10T20:30:16Z

chart/values.schema.json

                            "description": "The gitSync image tag.",
                            "type": "string",
-                            "default": "v4.4.2"
+                            "default": "v4.6.0"


I believe this should be in a separate PR as it is unrelated change for this one.

Miretpl · 2026-03-10T20:31:02Z

chart/values.schema.json

                        "credentialsSecret": {
-                            "description": "Name of a Secret containing the repo `GIT_SYNC_USERNAME` and `GIT_SYNC_PASSWORD`.",
+                            "description": "Name of a Secret containing `GIT_SYNC_USERNAME`, `GITSYNC_USERNAME`, `GIT_SYNC_PASSWORD`, and `GITSYNC_PASSWORD` keys. The password keys are mounted as files and used via `*_PASSWORD_FILE` env vars so rotated credentials can be re-read.",


You probably need to create a new field dedicated for the file as this breaks backward-compatibility.

sidshas03 · 2026-03-10T20:43:56Z

Thanks @Miretpl for the detailed review. I got your point on the backward compatibility part, and I will rework this accordingly. I will keep the existing credentialsSecret behaviour as it is, and instead add a separate field for the file based password support, so the current contract does not change. I will also remove the git-sync image tag bump from this PR and take it separately. And yes, I will add the corresponding change in the pod-template-file path used by KubernetesExecutor as well.

jscheffl · 2026-03-10T21:32:49Z

Note: We are currently planning and discussing a revamt to a chart v2 where we want to re-model some things and also drop some legacy. Would it be OK to "park" this PR, drop all git sync legacy support and use minimum 4.6.0 from then on and have this as a new feature? w/o all the backwards compatatibility complexity?

sidshas03 · 2026-03-10T22:01:54Z

Thanks for the guidance @jscheffl. I’m parking this PR as discussed. I’m interested to work on the chart v2 follow-up (git-sync >= 4.6.0) as well, please tag me when that issue/epic is created.

electrical · 2026-03-10T22:24:56Z

@jscheffl Do you have an estimation on the timeframe for the new Helm chart?
The current issue that this PR solves is not a major problem but would be nice to have it resolved to avoid crashing containers all the time :)

jscheffl · 2026-03-10T22:40:44Z

@electrical We had initial discussions, are currently collecting topics and will drop a DISCUSS to devlist once we have a (draft) plan.

Fix git-sync rotated credentials support in Helm chart

04d6287

sidshas03 requested review from bugraoz93, dstandish, hussein-awala, jedcunningham and jscheffl as code owners March 10, 2026 14:50

boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Mar 10, 2026

Fix airflow_aux git-sync credentials test for password file envs

a144940

Bump default git-sync image to v4.6.0 for password-file rotation support

2197b76

Miretpl suggested changes Mar 10, 2026

View reviewed changes

Make git-sync password-file mode opt-in and add pod-template support

573731d

jscheffl added this to the Airflow Helm Chart 2.0.0 milestone Mar 10, 2026

sidshas03 closed this Mar 11, 2026

sidshas03 reopened this Mar 11, 2026

	{{- if .Values.dags.gitSync.credentialsSecret }}
	{{- if and $localOrDagProcessorDisabled .Values.dags.gitSync.enabled .Values.dags.gitSync.credentialsSecret }}

	{{- if .Values.dags.gitSync.credentialsSecret }}
	{{- if and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled) }}

	{{/* Git credentials volume */}}
	{{/* Git credentials volume */}}

Conversation

sidshas03 commented Mar 10, 2026

Problem

What I changed

Why this fix

Testing

Uh oh!

electrical commented Mar 10, 2026

Uh oh!

sidshas03 commented Mar 10, 2026

Uh oh!

Miretpl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Miretpl Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

sidshas03 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl commented Mar 10, 2026

Uh oh!

sidshas03 commented Mar 10, 2026

Uh oh!

electrical commented Mar 10, 2026

Uh oh!

jscheffl commented Mar 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Miretpl left a comment •

edited

Loading

sidshas03 commented Mar 10, 2026 •

edited

Loading